{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tweet Cleaner\n",
    "\n",
    "I combined various text cleaning techniques from other notebooks into this one. The original source of each technique is credited alongside the relevant block of code. Features include:\n",
    "\n",
    " * Expands contractions (can't = cannot)\n",
    " * Expands tweet slang (OMG = Oh My God)\n",
    " * Removes emojis\n",
    " * Removes HTML tags\n",
    " * Removes URLs\n",
    " * Replaces repeated punctuation with a single character (Wow!!!!! = Wow!)\n",
    " * Removes special characters\n",
    " * Optionally removes common English stopwords\n",
    " * Optionally lemmatizes words, converts plural and conjugated words into a single root (corpora = corpus, rocks = rock)\n",
    " * Optionally corrects common mispellings\n",
    " * Uses parallelized code to quickly process a dataframe\n",
    "\n",
    "The spell check method sometimes garbles text, and greatly slows down execution, so it is not run by default. Stopword removal and lemmatization are included, but off by default. Skip to the \"Example usage\" heading to see the end result. **If you have any ideas for improving this notebook, please let me know in the comments!**\n",
    "\n",
    "Check out my other notebook [GPT-2: fake real disasters [data augmentation]](https://www.kaggle.com/jdparsons/gpt-2-fake-real-disasters-data-augmentation) where I send these cleaned tweets to the infamous GPT-2 model in order to augment the training dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting pyspellchecker\r\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/93/24/9a570f49dfefc16e9ce1f483bb2d5bff701b95094e051db502e3c11f5092/pyspellchecker-0.5.3-py2.py3-none-any.whl (1.9MB)\r\n",
      "\u001b[K     |████████████████████████████████| 1.9MB 3.4MB/s \r\n",
      "\u001b[?25hInstalling collected packages: pyspellchecker\r\n",
      "Successfully installed pyspellchecker-0.5.3\r\n",
      "/kaggle/input/real-or-not-gpt2-augmented-training-tweets/train_df_GPT2_augmented.csv\n",
      "/kaggle/input/nlp-getting-started/test.csv\n",
      "/kaggle/input/nlp-getting-started/train.csv\n",
      "/kaggle/input/nlp-getting-started/sample_submission.csv\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import re\n",
    "from nltk.tokenize import word_tokenize\n",
    "!pip install pyspellchecker\n",
    "from spellchecker import SpellChecker\n",
    "import time\n",
    "from multiprocessing import  Pool\n",
    "from nltk.stem import WordNetLemmatizer \n",
    "\n",
    "import os\n",
    "for dirname, _, filenames in os.walk('/kaggle/input'):\n",
    "    for filename in filenames:\n",
    "        print(os.path.join(dirname, filename))\n",
    "\n",
    "# set pandas preview to use full width of browser\n",
    "pd.set_option('display.max_columns', None)\n",
    "pd.set_option('display.expand_frame_repr', False)\n",
    "pd.set_option('max_colwidth', -1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0",
    "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a"
   },
   "outputs": [],
   "source": [
    "train = pd.read_csv('../input/nlp-getting-started/train.csv')\n",
    "test = pd.read_csv('../input/nlp-getting-started/test.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Main clean functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://stackoverflow.com/a/34682849\n",
    "def untokenize(words):\n",
    "    \"\"\"Untokenizing a text undoes the tokenizing operation, restoring\n",
    "    punctuation and spaces to the places that people expect them to be.\n",
    "    Ideally, `untokenize(tokenize(text))` should be identical to `text`,\n",
    "    except for line breaks.\n",
    "    \"\"\"\n",
    "    text = ' '.join(words)\n",
    "    step1 = text.replace(\"`` \", '\"').replace(\" ''\", '\"').replace('. . .', '...')\n",
    "    step2 = step1.replace(\" ( \", \" (\").replace(\" ) \", \") \")\n",
    "    step3 = re.sub(r' ([.,:;?!%]+)([ \\'\"`])', r\"\\1\\2\", step2)\n",
    "    step4 = re.sub(r' ([.,:;?!%]+)$', r\"\\1\", step3)\n",
    "    step5 = step4.replace(\" '\", \"'\").replace(\" n't\", \"n't\").replace(\n",
    "        \"can not\", \"cannot\")\n",
    "    step6 = step5.replace(\" ` \", \" '\")\n",
    "    return step6.strip()\n",
    "\n",
    "\n",
    "# https://stackoverflow.com/a/47091490\n",
    "def decontracted(phrase):\n",
    "    \"\"\"Convert contractions like \"can't\" into \"can not\"\n",
    "    \"\"\"\n",
    "    # specific\n",
    "    phrase = re.sub(r\"won\\'t\", \"will not\", phrase)\n",
    "    phrase = re.sub(r\"can\\'t\", \"can not\", phrase)\n",
    "\n",
    "    # general\n",
    "    #phrase = re.sub(r\"n't\", \" not\", phrase) # resulted in \"ca not\" when sentence started with \"can't\"\n",
    "    phrase = re.sub(r\"\\'re\", \" are\", phrase)\n",
    "    phrase = re.sub(r\"\\'s\", \" is\", phrase)\n",
    "    phrase = re.sub(r\"\\'d\", \" would\", phrase)\n",
    "    phrase = re.sub(r\"\\'ll\", \" will\", phrase)\n",
    "    phrase = re.sub(r\"\\'t\", \" not\", phrase)\n",
    "    phrase = re.sub(r\"\\'ve\", \" have\", phrase)\n",
    "    phrase = re.sub(r\"\\'m\", \" am\", phrase)\n",
    "    return phrase\n",
    "\n",
    "\n",
    "# https://github.com/rishabhverma17/sms_slang_translator/blob/master/slang.txt\n",
    "slang_abbrev_dict = {\n",
    "    'AFAIK': 'As Far As I Know',\n",
    "    'AFK': 'Away From Keyboard',\n",
    "    'ASAP': 'As Soon As Possible',\n",
    "    'ATK': 'At The Keyboard',\n",
    "    'ATM': 'At The Moment',\n",
    "    'A3': 'Anytime, Anywhere, Anyplace',\n",
    "    'BAK': 'Back At Keyboard',\n",
    "    'BBL': 'Be Back Later',\n",
    "    'BBS': 'Be Back Soon',\n",
    "    'BFN': 'Bye For Now',\n",
    "    'B4N': 'Bye For Now',\n",
    "    'BRB': 'Be Right Back',\n",
    "    'BRT': 'Be Right There',\n",
    "    'BTW': 'By The Way',\n",
    "    'B4': 'Before',\n",
    "    'B4N': 'Bye For Now',\n",
    "    'CU': 'See You',\n",
    "    'CUL8R': 'See You Later',\n",
    "    'CYA': 'See You',\n",
    "    'FAQ': 'Frequently Asked Questions',\n",
    "    'FC': 'Fingers Crossed',\n",
    "    'FWIW': 'For What It\\'s Worth',\n",
    "    'FYI': 'For Your Information',\n",
    "    'GAL': 'Get A Life',\n",
    "    'GG': 'Good Game',\n",
    "    'GN': 'Good Night',\n",
    "    'GMTA': 'Great Minds Think Alike',\n",
    "    'GR8': 'Great!',\n",
    "    'G9': 'Genius',\n",
    "    'IC': 'I See',\n",
    "    'ICQ': 'I Seek you',\n",
    "    'ILU': 'I Love You',\n",
    "    'IMHO': 'In My Humble Opinion',\n",
    "    'IMO': 'In My Opinion',\n",
    "    'IOW': 'In Other Words',\n",
    "    'IRL': 'In Real Life',\n",
    "    'KISS': 'Keep It Simple, Stupid',\n",
    "    'LDR': 'Long Distance Relationship',\n",
    "    'LMAO': 'Laugh My Ass Off',\n",
    "    'LOL': 'Laughing Out Loud',\n",
    "    'LTNS': 'Long Time No See',\n",
    "    'L8R': 'Later',\n",
    "    'MTE': 'My Thoughts Exactly',\n",
    "    'M8': 'Mate',\n",
    "    'NRN': 'No Reply Necessary',\n",
    "    'OIC': 'Oh I See',\n",
    "    'OMG': 'Oh My God',\n",
    "    'PITA': 'Pain In The Ass',\n",
    "    'PRT': 'Party',\n",
    "    'PRW': 'Parents Are Watching',\n",
    "    'QPSA?': 'Que Pasa?',\n",
    "    'ROFL': 'Rolling On The Floor Laughing',\n",
    "    'ROFLOL': 'Rolling On The Floor Laughing Out Loud',\n",
    "    'ROTFLMAO': 'Rolling On The Floor Laughing My Ass Off',\n",
    "    'SK8': 'Skate',\n",
    "    'STATS': 'Your sex and age',\n",
    "    'ASL': 'Age, Sex, Location',\n",
    "    'THX': 'Thank You',\n",
    "    'TTFN': 'Ta-Ta For Now!',\n",
    "    'TTYL': 'Talk To You Later',\n",
    "    'U': 'You',\n",
    "    'U2': 'You Too',\n",
    "    'U4E': 'Yours For Ever',\n",
    "    'WB': 'Welcome Back',\n",
    "    'WTF': 'What The Fuck',\n",
    "    'WTG': 'Way To Go!',\n",
    "    'WUF': 'Where Are You From?',\n",
    "    'W8': 'Wait',\n",
    "    '7K': 'Sick:-D Laugher'\n",
    "}\n",
    "\n",
    "\n",
    "def unslang(text):\n",
    "    \"\"\"Converts text like \"OMG\" into \"Oh my God\"\n",
    "    \"\"\"\n",
    "    if text.upper() in slang_abbrev_dict.keys():\n",
    "        return slang_abbrev_dict[text.upper()]\n",
    "    else:\n",
    "        return text\n",
    "\n",
    "\n",
    "# https://gist.github.com/sebleier/554280\n",
    "stopwords = [\n",
    "    \"a\", \"about\", \"above\", \"after\", \"again\", \"against\", \"ain\", \"all\", \"am\",\n",
    "    \"an\", \"and\", \"any\", \"are\", \"aren\", \"aren't\", \"as\", \"at\", \"be\", \"because\",\n",
    "    \"been\", \"before\", \"being\", \"below\", \"between\", \"both\", \"but\", \"by\", \"can\",\n",
    "    \"couldn\", \"couldn't\", \"d\", \"did\", \"didn\", \"didn't\", \"do\", \"does\", \"doesn\",\n",
    "    \"doesn't\", \"doing\", \"don\", \"don't\", \"down\", \"during\", \"each\", \"few\", \"for\",\n",
    "    \"from\", \"further\", \"had\", \"hadn\", \"hadn't\", \"has\", \"hasn\", \"hasn't\", \"have\",\n",
    "    \"haven\", \"haven't\", \"having\", \"he\", \"her\", \"here\", \"hers\", \"herself\", \"him\",\n",
    "    \"himself\", \"his\", \"how\", \"i\", \"if\", \"in\", \"into\", \"is\", \"isn\", \"isn't\",\n",
    "    \"it\", \"it's\", \"its\", \"itself\", \"just\", \"ll\", \"m\", \"ma\", \"me\", \"mightn\",\n",
    "    \"mightn't\", \"more\", \"most\", \"mustn\", \"mustn't\", \"my\", \"myself\", \"needn\",\n",
    "    \"needn't\", \"no\", \"nor\", \"not\", \"now\", \"o\", \"of\", \"off\", \"on\", \"once\",\n",
    "    \"only\", \"or\", \"other\", \"our\", \"ours\", \"ourselves\", \"out\", \"over\", \"own\",\n",
    "    \"re\", \"s\", \"same\", \"shan\", \"shan't\", \"she\", \"she's\", \"should\", \"should've\",\n",
    "    \"shouldn\", \"shouldn't\", \"so\", \"some\", \"such\", \"t\", \"than\", \"that\",\n",
    "    \"that'll\", \"the\", \"their\", \"theirs\", \"them\", \"themselves\", \"then\", \"there\",\n",
    "    \"these\", \"they\", \"this\", \"those\", \"through\", \"to\", \"too\", \"under\", \"until\",\n",
    "    \"up\", \"ve\", \"very\", \"was\", \"wasn\", \"wasn't\", \"we\", \"were\", \"weren\",\n",
    "    \"weren't\", \"what\", \"when\", \"where\", \"which\", \"while\", \"who\", \"whom\", \"why\",\n",
    "    \"will\", \"with\", \"won\", \"won't\", \"wouldn\", \"wouldn't\", \"y\", \"you\", \"you'd\",\n",
    "    \"you'll\", \"you're\", \"you've\", \"your\", \"yours\", \"yourself\", \"yourselves\",\n",
    "    \"could\", \"he'd\", \"he'll\", \"he's\", \"here's\", \"how's\", \"i'd\", \"i'll\", \"i'm\",\n",
    "    \"i've\", \"let's\", \"ought\", \"she'd\", \"she'll\", \"that's\", \"there's\", \"they'd\",\n",
    "    \"they'll\", \"they're\", \"they've\", \"we'd\", \"we'll\", \"we're\", \"we've\",\n",
    "    \"what's\", \"when's\", \"where's\", \"who's\", \"why's\", \"would\", \"able\", \"abst\",\n",
    "    \"accordance\", \"according\", \"accordingly\", \"across\", \"act\", \"actually\",\n",
    "    \"added\", \"adj\", \"affected\", \"affecting\", \"affects\", \"afterwards\", \"ah\",\n",
    "    \"almost\", \"alone\", \"along\", \"already\", \"also\", \"although\", \"always\",\n",
    "    \"among\", \"amongst\", \"announce\", \"another\", \"anybody\", \"anyhow\", \"anymore\",\n",
    "    \"anyone\", \"anything\", \"anyway\", \"anyways\", \"anywhere\", \"apparently\",\n",
    "    \"approximately\", \"arent\", \"arise\", \"around\", \"aside\", \"ask\", \"asking\",\n",
    "    \"auth\", \"available\", \"away\", \"awfully\", \"b\", \"back\", \"became\", \"become\",\n",
    "    \"becomes\", \"becoming\", \"beforehand\", \"begin\", \"beginning\", \"beginnings\",\n",
    "    \"begins\", \"behind\", \"believe\", \"beside\", \"besides\", \"beyond\", \"biol\",\n",
    "    \"brief\", \"briefly\", \"c\", \"ca\", \"came\", \"cannot\", \"can't\", \"cause\", \"causes\",\n",
    "    \"certain\", \"certainly\", \"co\", \"com\", \"come\", \"comes\", \"contain\",\n",
    "    \"containing\", \"contains\", \"couldnt\", \"date\", \"different\", \"done\",\n",
    "    \"downwards\", \"due\", \"e\", \"ed\", \"edu\", \"effect\", \"eg\", \"eight\", \"eighty\",\n",
    "    \"either\", \"else\", \"elsewhere\", \"end\", \"ending\", \"enough\", \"especially\",\n",
    "    \"et\", \"etc\", \"even\", \"ever\", \"every\", \"everybody\", \"everyone\", \"everything\",\n",
    "    \"everywhere\", \"ex\", \"except\", \"f\", \"far\", \"ff\", \"fifth\", \"first\", \"five\",\n",
    "    \"fix\", \"followed\", \"following\", \"follows\", \"former\", \"formerly\", \"forth\",\n",
    "    \"found\", \"four\", \"furthermore\", \"g\", \"gave\", \"get\", \"gets\", \"getting\",\n",
    "    \"give\", \"given\", \"gives\", \"giving\", \"go\", \"goes\", \"gone\", \"got\", \"gotten\",\n",
    "    \"h\", \"happens\", \"hardly\", \"hed\", \"hence\", \"hereafter\", \"hereby\", \"herein\",\n",
    "    \"heres\", \"hereupon\", \"hes\", \"hi\", \"hid\", \"hither\", \"home\", \"howbeit\",\n",
    "    \"however\", \"hundred\", \"id\", \"ie\", \"im\", \"immediate\", \"immediately\",\n",
    "    \"importance\", \"important\", \"inc\", \"indeed\", \"index\", \"information\",\n",
    "    \"instead\", \"invention\", \"inward\", \"itd\", \"it'll\", \"j\", \"k\", \"keep\", \"keeps\",\n",
    "    \"kept\", \"kg\", \"km\", \"know\", \"known\", \"knows\", \"l\", \"largely\", \"last\",\n",
    "    \"lately\", \"later\", \"latter\", \"latterly\", \"least\", \"less\", \"lest\", \"let\",\n",
    "    \"lets\", \"like\", \"liked\", \"likely\", \"line\", \"little\", \"'ll\", \"look\",\n",
    "    \"looking\", \"looks\", \"ltd\", \"made\", \"mainly\", \"make\", \"makes\", \"many\", \"may\",\n",
    "    \"maybe\", \"mean\", \"means\", \"meantime\", \"meanwhile\", \"merely\", \"mg\", \"might\",\n",
    "    \"million\", \"miss\", \"ml\", \"moreover\", \"mostly\", \"mr\", \"mrs\", \"much\", \"mug\",\n",
    "    \"must\", \"n\", \"na\", \"name\", \"namely\", \"nay\", \"nd\", \"near\", \"nearly\",\n",
    "    \"necessarily\", \"necessary\", \"need\", \"needs\", \"neither\", \"never\",\n",
    "    \"nevertheless\", \"new\", \"next\", \"nine\", \"ninety\", \"nobody\", \"non\", \"none\",\n",
    "    \"nonetheless\", \"noone\", \"normally\", \"nos\", \"noted\", \"nothing\", \"nowhere\",\n",
    "    \"obtain\", \"obtained\", \"obviously\", \"often\", \"oh\", \"ok\", \"okay\", \"old\",\n",
    "    \"omitted\", \"one\", \"ones\", \"onto\", \"ord\", \"others\", \"otherwise\", \"outside\",\n",
    "    \"overall\", \"owing\", \"p\", \"page\", \"pages\", \"part\", \"particular\",\n",
    "    \"particularly\", \"past\", \"per\", \"perhaps\", \"placed\", \"please\", \"plus\",\n",
    "    \"poorly\", \"possible\", \"possibly\", \"potentially\", \"pp\", \"predominantly\",\n",
    "    \"present\", \"previously\", \"primarily\", \"probably\", \"promptly\", \"proud\",\n",
    "    \"provides\", \"put\", \"q\", \"que\", \"quickly\", \"quite\", \"qv\", \"r\", \"ran\",\n",
    "    \"rather\", \"rd\", \"readily\", \"really\", \"recent\", \"recently\", \"ref\", \"refs\",\n",
    "    \"regarding\", \"regardless\", \"regards\", \"related\", \"relatively\", \"research\",\n",
    "    \"respectively\", \"resulted\", \"resulting\", \"results\", \"right\", \"run\", \"said\",\n",
    "    \"saw\", \"say\", \"saying\", \"says\", \"sec\", \"section\", \"see\", \"seeing\", \"seem\",\n",
    "    \"seemed\", \"seeming\", \"seems\", \"seen\", \"self\", \"selves\", \"sent\", \"seven\",\n",
    "    \"several\", \"shall\", \"shed\", \"shes\", \"show\", \"showed\", \"shown\", \"showns\",\n",
    "    \"shows\", \"significant\", \"significantly\", \"similar\", \"similarly\", \"since\",\n",
    "    \"six\", \"slightly\", \"somebody\", \"somehow\", \"someone\", \"somethan\",\n",
    "    \"something\", \"sometime\", \"sometimes\", \"somewhat\", \"somewhere\", \"soon\",\n",
    "    \"sorry\", \"specifically\", \"specified\", \"specify\", \"specifying\", \"still\",\n",
    "    \"stop\", \"strongly\", \"sub\", \"substantially\", \"successfully\", \"sufficiently\",\n",
    "    \"suggest\", \"sup\", \"sure\", \"take\", \"taken\", \"taking\", \"tell\", \"tends\", \"th\",\n",
    "    \"thank\", \"thanks\", \"thanx\", \"thats\", \"that've\", \"thence\", \"thereafter\",\n",
    "    \"thereby\", \"thered\", \"therefore\", \"therein\", \"there'll\", \"thereof\",\n",
    "    \"therere\", \"theres\", \"thereto\", \"thereupon\", \"there've\", \"theyd\", \"theyre\",\n",
    "    \"think\", \"thou\", \"though\", \"thoughh\", \"thousand\", \"throug\", \"throughout\",\n",
    "    \"thru\", \"thus\", \"til\", \"tip\", \"together\", \"took\", \"toward\", \"towards\",\n",
    "    \"tried\", \"tries\", \"truly\", \"try\", \"trying\", \"ts\", \"twice\", \"two\", \"u\", \"un\",\n",
    "    \"unfortunately\", \"unless\", \"unlike\", \"unlikely\", \"unto\", \"upon\", \"ups\",\n",
    "    \"us\", \"use\", \"used\", \"useful\", \"usefully\", \"usefulness\", \"uses\", \"using\",\n",
    "    \"usually\", \"v\", \"value\", \"various\", \"'ve\", \"via\", \"viz\", \"vol\", \"vols\",\n",
    "    \"vs\", \"w\", \"want\", \"wants\", \"wasnt\", \"way\", \"wed\", \"welcome\", \"went\",\n",
    "    \"werent\", \"whatever\", \"what'll\", \"whats\", \"whence\", \"whenever\",\n",
    "    \"whereafter\", \"whereas\", \"whereby\", \"wherein\", \"wheres\", \"whereupon\",\n",
    "    \"wherever\", \"whether\", \"whim\", \"whither\", \"whod\", \"whoever\", \"whole\",\n",
    "    \"who'll\", \"whomever\", \"whos\", \"whose\", \"widely\", \"willing\", \"wish\",\n",
    "    \"within\", \"without\", \"wont\", \"words\", \"world\", \"wouldnt\", \"www\", \"x\", \"yes\",\n",
    "    \"yet\", \"youd\", \"youre\", \"z\", \"zero\", \"a's\", \"ain't\", \"allow\", \"allows\",\n",
    "    \"apart\", \"appear\", \"appreciate\", \"appropriate\", \"associated\", \"best\",\n",
    "    \"better\", \"c'mon\", \"c's\", \"cant\", \"changes\", \"clearly\", \"concerning\",\n",
    "    \"consequently\", \"consider\", \"considering\", \"corresponding\", \"course\",\n",
    "    \"currently\", \"definitely\", \"described\", \"despite\", \"entirely\", \"exactly\",\n",
    "    \"example\", \"going\", \"greetings\", \"hello\", \"help\", \"hopefully\", \"ignored\",\n",
    "    \"inasmuch\", \"indicate\", \"indicated\", \"indicates\", \"inner\", \"insofar\",\n",
    "    \"it'd\", \"keep\", \"keeps\", \"novel\", \"presumably\", \"reasonably\", \"second\",\n",
    "    \"secondly\", \"sensible\", \"serious\", \"seriously\", \"sure\", \"t's\", \"third\",\n",
    "    \"thorough\", \"thoroughly\", \"three\", \"well\", \"wonder\", \"a\", \"about\", \"above\",\n",
    "    \"above\", \"across\", \"after\", \"afterwards\", \"again\", \"against\", \"all\",\n",
    "    \"almost\", \"alone\", \"along\", \"already\", \"also\", \"although\", \"always\", \"am\",\n",
    "    \"among\", \"amongst\", \"amoungst\", \"amount\", \"an\", \"and\", \"another\", \"any\",\n",
    "    \"anyhow\", \"anyone\", \"anything\", \"anyway\", \"anywhere\", \"are\", \"around\", \"as\",\n",
    "    \"at\", \"back\", \"be\", \"became\", \"because\", \"become\", \"becomes\", \"becoming\",\n",
    "    \"been\", \"before\", \"beforehand\", \"behind\", \"being\", \"below\", \"beside\",\n",
    "    \"besides\", \"between\", \"beyond\", \"bill\", \"both\", \"bottom\", \"but\", \"by\",\n",
    "    \"call\", \"can\", \"cannot\", \"cant\", \"co\", \"con\", \"could\", \"couldnt\", \"cry\",\n",
    "    \"de\", \"describe\", \"detail\", \"do\", \"done\", \"down\", \"due\", \"during\", \"each\",\n",
    "    \"eg\", \"eight\", \"either\", \"eleven\", \"else\", \"elsewhere\", \"empty\", \"enough\",\n",
    "    \"etc\", \"even\", \"ever\", \"every\", \"everyone\", \"everything\", \"everywhere\",\n",
    "    \"except\", \"few\", \"fifteen\", \"fify\", \"fill\", \"find\", \"fire\", \"first\", \"five\",\n",
    "    \"for\", \"former\", \"formerly\", \"forty\", \"found\", \"four\", \"from\", \"front\",\n",
    "    \"full\", \"further\", \"get\", \"give\", \"go\", \"had\", \"has\", \"hasnt\", \"have\", \"he\",\n",
    "    \"hence\", \"her\", \"here\", \"hereafter\", \"hereby\", \"herein\", \"hereupon\", \"hers\",\n",
    "    \"herself\", \"him\", \"himself\", \"his\", \"how\", \"however\", \"hundred\", \"ie\", \"if\",\n",
    "    \"in\", \"inc\", \"indeed\", \"interest\", \"into\", \"is\", \"it\", \"its\", \"itself\",\n",
    "    \"keep\", \"last\", \"latter\", \"latterly\", \"least\", \"less\", \"ltd\", \"made\",\n",
    "    \"many\", \"may\", \"me\", \"meanwhile\", \"might\", \"mill\", \"mine\", \"more\",\n",
    "    \"moreover\", \"most\", \"mostly\", \"move\", \"much\", \"must\", \"my\", \"myself\",\n",
    "    \"name\", \"namely\", \"neither\", \"never\", \"nevertheless\", \"next\", \"nine\", \"no\",\n",
    "    \"nobody\", \"none\", \"noone\", \"nor\", \"not\", \"nothing\", \"now\", \"nowhere\", \"of\",\n",
    "    \"off\", \"often\", \"on\", \"once\", \"one\", \"only\", \"onto\", \"or\", \"other\",\n",
    "    \"others\", \"otherwise\", \"our\", \"ours\", \"ourselves\", \"out\", \"over\", \"own\",\n",
    "    \"part\", \"per\", \"perhaps\", \"please\", \"put\", \"rather\", \"re\", \"same\", \"see\",\n",
    "    \"seem\", \"seemed\", \"seeming\", \"seems\", \"serious\", \"several\", \"she\", \"should\",\n",
    "    \"show\", \"side\", \"since\", \"sincere\", \"six\", \"sixty\", \"so\", \"some\", \"somehow\",\n",
    "    \"someone\", \"something\", \"sometime\", \"sometimes\", \"somewhere\", \"still\",\n",
    "    \"such\", \"system\", \"take\", \"ten\", \"than\", \"that\", \"the\", \"their\", \"them\",\n",
    "    \"themselves\", \"then\", \"thence\", \"there\", \"thereafter\", \"thereby\",\n",
    "    \"therefore\", \"therein\", \"thereupon\", \"these\", \"they\", \"thickv\", \"thin\",\n",
    "    \"third\", \"this\", \"those\", \"though\", \"three\", \"through\", \"throughout\",\n",
    "    \"thru\", \"thus\", \"to\", \"together\", \"too\", \"top\", \"toward\", \"towards\",\n",
    "    \"twelve\", \"twenty\", \"two\", \"un\", \"under\", \"until\", \"up\", \"upon\", \"us\",\n",
    "    \"very\", \"via\", \"was\", \"we\", \"well\", \"were\", \"what\", \"whatever\", \"when\",\n",
    "    \"whence\", \"whenever\", \"where\", \"whereafter\", \"whereas\", \"whereby\",\n",
    "    \"wherein\", \"whereupon\", \"wherever\", \"whether\", \"which\", \"while\", \"whither\",\n",
    "    \"who\", \"whoever\", \"whole\", \"whom\", \"whose\", \"why\", \"will\", \"with\", \"within\",\n",
    "    \"without\", \"would\", \"yet\", \"you\", \"your\", \"yours\", \"yourself\", \"yourselves\",\n",
    "    \"the\", \"a\", \"b\", \"c\", \"d\", \"e\", \"f\", \"g\", \"h\", \"i\", \"j\", \"k\", \"l\", \"m\", \"n\",\n",
    "    \"o\", \"p\", \"q\", \"r\", \"s\", \"t\", \"u\", \"v\", \"w\", \"x\", \"y\", \"z\", \"A\", \"B\", \"C\",\n",
    "    \"D\", \"E\", \"F\", \"G\", \"H\", \"I\", \"J\", \"K\", \"L\", \"M\", \"N\", \"O\", \"P\", \"Q\", \"R\",\n",
    "    \"S\", \"T\", \"U\", \"V\", \"W\", \"X\", \"Y\", \"Z\", \"co\", \"op\", \"research-articl\",\n",
    "    \"pagecount\", \"cit\", \"ibid\", \"les\", \"le\", \"au\", \"que\", \"est\", \"pas\", \"vol\",\n",
    "    \"el\", \"los\", \"pp\", \"u201d\", \"well-b\", \"http\", \"volumtype\", \"par\", \"0o\",\n",
    "    \"0s\", \"3a\", \"3b\", \"3d\", \"6b\", \"6o\", \"a1\", \"a2\", \"a3\", \"a4\", \"ab\", \"ac\",\n",
    "    \"ad\", \"ae\", \"af\", \"ag\", \"aj\", \"al\", \"an\", \"ao\", \"ap\", \"ar\", \"av\", \"aw\",\n",
    "    \"ax\", \"ay\", \"az\", \"b1\", \"b2\", \"b3\", \"ba\", \"bc\", \"bd\", \"be\", \"bi\", \"bj\",\n",
    "    \"bk\", \"bl\", \"bn\", \"bp\", \"br\", \"bs\", \"bt\", \"bu\", \"bx\", \"c1\", \"c2\", \"c3\",\n",
    "    \"cc\", \"cd\", \"ce\", \"cf\", \"cg\", \"ch\", \"ci\", \"cj\", \"cl\", \"cm\", \"cn\", \"cp\",\n",
    "    \"cq\", \"cr\", \"cs\", \"ct\", \"cu\", \"cv\", \"cx\", \"cy\", \"cz\", \"d2\", \"da\", \"dc\",\n",
    "    \"dd\", \"de\", \"df\", \"di\", \"dj\", \"dk\", \"dl\", \"do\", \"dp\", \"dr\", \"ds\", \"dt\",\n",
    "    \"du\", \"dx\", \"dy\", \"e2\", \"e3\", \"ea\", \"ec\", \"ed\", \"ee\", \"ef\", \"ei\", \"ej\",\n",
    "    \"el\", \"em\", \"en\", \"eo\", \"ep\", \"eq\", \"er\", \"es\", \"et\", \"eu\", \"ev\", \"ex\",\n",
    "    \"ey\", \"f2\", \"fa\", \"fc\", \"ff\", \"fi\", \"fj\", \"fl\", \"fn\", \"fo\", \"fr\", \"fs\",\n",
    "    \"ft\", \"fu\", \"fy\", \"ga\", \"ge\", \"gi\", \"gj\", \"gl\", \"go\", \"gr\", \"gs\", \"gy\",\n",
    "    \"h2\", \"h3\", \"hh\", \"hi\", \"hj\", \"ho\", \"hr\", \"hs\", \"hu\", \"hy\", \"i\", \"i2\", \"i3\",\n",
    "    \"i4\", \"i6\", \"i7\", \"i8\", \"ia\", \"ib\", \"ic\", \"ie\", \"ig\", \"ih\", \"ii\", \"ij\",\n",
    "    \"il\", \"in\", \"io\", \"ip\", \"iq\", \"ir\", \"iv\", \"ix\", \"iy\", \"iz\", \"jj\", \"jr\",\n",
    "    \"js\", \"jt\", \"ju\", \"ke\", \"kg\", \"kj\", \"km\", \"ko\", \"l2\", \"la\", \"lb\", \"lc\",\n",
    "    \"lf\", \"lj\", \"ln\", \"lo\", \"lr\", \"ls\", \"lt\", \"m2\", \"ml\", \"mn\", \"mo\", \"ms\",\n",
    "    \"mt\", \"mu\", \"n2\", \"nc\", \"nd\", \"ne\", \"ng\", \"ni\", \"nj\", \"nl\", \"nn\", \"nr\",\n",
    "    \"ns\", \"nt\", \"ny\", \"oa\", \"ob\", \"oc\", \"od\", \"of\", \"og\", \"oi\", \"oj\", \"ol\",\n",
    "    \"om\", \"on\", \"oo\", \"oq\", \"or\", \"os\", \"ot\", \"ou\", \"ow\", \"ox\", \"oz\", \"p1\",\n",
    "    \"p2\", \"p3\", \"pc\", \"pd\", \"pe\", \"pf\", \"ph\", \"pi\", \"pj\", \"pk\", \"pl\", \"pm\",\n",
    "    \"pn\", \"po\", \"pq\", \"pr\", \"ps\", \"pt\", \"pu\", \"py\", \"qj\", \"qu\", \"r2\", \"ra\",\n",
    "    \"rc\", \"rd\", \"rf\", \"rh\", \"ri\", \"rj\", \"rl\", \"rm\", \"rn\", \"ro\", \"rq\", \"rr\",\n",
    "    \"rs\", \"rt\", \"ru\", \"rv\", \"ry\", \"s2\", \"sa\", \"sc\", \"sd\", \"se\", \"sf\", \"si\",\n",
    "    \"sj\", \"sl\", \"sm\", \"sn\", \"sp\", \"sq\", \"sr\", \"ss\", \"st\", \"sy\", \"sz\", \"t1\",\n",
    "    \"t2\", \"t3\", \"tb\", \"tc\", \"td\", \"te\", \"tf\", \"th\", \"ti\", \"tj\", \"tl\", \"tm\",\n",
    "    \"tn\", \"tp\", \"tq\", \"tr\", \"ts\", \"tt\", \"tv\", \"tx\", \"ue\", \"ui\", \"uj\", \"uk\",\n",
    "    \"um\", \"un\", \"uo\", \"ur\", \"ut\", \"va\", \"wa\", \"vd\", \"wi\", \"vj\", \"vo\", \"wo\",\n",
    "    \"vq\", \"vt\", \"vu\", \"x1\", \"x2\", \"x3\", \"xf\", \"xi\", \"xj\", \"xk\", \"xl\", \"xn\",\n",
    "    \"xo\", \"xs\", \"xt\", \"xv\", \"xx\", \"y2\", \"yj\", \"yl\", \"yr\", \"ys\", \"yt\", \"zi\", \"zz\"\n",
    "]\n",
    "\n",
    "\n",
    "# Reference : https://gist.github.com/slowkow/7a7f61f495e3dbb7e3d767f97bd7304b\n",
    "def remove_emoji(text):\n",
    "    emoji_pattern = re.compile(\n",
    "        \"[\"\n",
    "        u\"\\U0001F600-\\U0001F64F\"  # emoticons\n",
    "        u\"\\U0001F300-\\U0001F5FF\"  # symbols & pictographs\n",
    "        u\"\\U0001F680-\\U0001F6FF\"  # transport & map symbols\n",
    "        u\"\\U0001F1E0-\\U0001F1FF\"  # flags (iOS)\n",
    "        u\"\\U00002702-\\U000027B0\"\n",
    "        u\"\\U000024C2-\\U0001F251\"\n",
    "        \"]+\",\n",
    "        flags=re.UNICODE)\n",
    "    return emoji_pattern.sub(r'', text)\n",
    "\n",
    "\n",
    "# from: https://www.kaggle.com/shahules/basic-eda-cleaning-and-glove\n",
    "# maybe a bug, it removes question marks?\n",
    "spell = SpellChecker()\n",
    "\n",
    "def correct_spellings(text):\n",
    "    corrected_text = []\n",
    "    misspelled_words = spell.unknown(text.split())\n",
    "    for word in text.split():\n",
    "        if word in misspelled_words:\n",
    "            corrected_text.append(spell.correction(word))\n",
    "        else:\n",
    "            corrected_text.append(word)\n",
    "    return \" \".join(corrected_text)\n",
    "\n",
    "def remove_urls(text):\n",
    "    text = clean(r\"http\\S+\", text)\n",
    "    text = clean(r\"www\\S+\", text)\n",
    "    text = clean(r\"pic.twitter.com\\S+\", text)\n",
    "\n",
    "    return text\n",
    "\n",
    "def clean(reg_exp, text):\n",
    "    text = re.sub(reg_exp, \" \", text)\n",
    "\n",
    "    # replace multiple spaces with one.\n",
    "    text = re.sub('\\s{2,}', ' ', text)\n",
    "\n",
    "    return text\n",
    "\n",
    "lemmatizer = WordNetLemmatizer()\n",
    "\n",
    "def clean_all(t, correct_spelling=False, remove_stopwords=False, lemmatize=False):\n",
    "    \n",
    "    # first do bulk cleanup on tokens that don't depend on word tokenization\n",
    "\n",
    "    # remove xml tags\n",
    "    t = clean(r\"<[^>]+>\", t)\n",
    "    t = clean(r\"&lt;\", t)\n",
    "    t = clean(r\"&gt;\", t)\n",
    "\n",
    "    # remove URLs\n",
    "    t = remove_urls(t)\n",
    "\n",
    "    # https://stackoverflow.com/a/35041925\n",
    "    # replace multiple punctuation with single. Ex: !?!?!? would become ?\n",
    "    t = clean(r'[\\?\\.\\!]+(?=[\\?\\.\\!])', t)\n",
    "\n",
    "    t = remove_emoji(t)\n",
    "\n",
    "    # expand common contractions like \"I'm\" \"he'll\"\n",
    "    t = decontracted(t)\n",
    "\n",
    "    # now remove/expand bad patterns per word\n",
    "    words = word_tokenize(t)\n",
    "\n",
    "    # remove stopwords\n",
    "    if remove_stopwords is True:\n",
    "        words = [w for w in words if not w in stopwords]\n",
    "\n",
    "    clean_words = []\n",
    "\n",
    "    for w in words:\n",
    "        # normalize punctuation\n",
    "        w = re.sub(r'&', 'and', w)\n",
    "\n",
    "        # expand slang like OMG = Oh my God\n",
    "        w = unslang(w)\n",
    "\n",
    "        if lemmatize is True:\n",
    "            w = lemmatizer.lemmatize(w)\n",
    "        \n",
    "        clean_words.append(w)\n",
    "\n",
    "    # join the words back into a full string\n",
    "    t = untokenize(clean_words)\n",
    "\n",
    "    if correct_spelling is True:\n",
    "        # this resulted in lots of lost punctuation - omitting for now. Also greatly speeds things up\n",
    "        t = correct_spellings(t)\n",
    "\n",
    "    # finally, remove any non ascii and special characters that made it through\n",
    "    t = clean(r\"[^A-Za-z0-9\\.\\'!\\?,\\$]\", t)\n",
    "\n",
    "    return t\n",
    "\n",
    "\n",
    "def clean_dataframe(df, correct_spelling=False, remove_stopwords=False):\n",
    "    df['clean'] = df.apply(lambda x: clean_all(x['text'], correct_spelling=correct_spelling, remove_stopwords=remove_stopwords), axis=1)\n",
    "\n",
    "    return df\n",
    "\n",
    "\n",
    "# https://towardsdatascience.com/make-your-own-super-pandas-using-multiproc-1c04f41944a1\n",
    "def parallelize_dataframe(\n",
    "        df, func, n_cores=2):  # I think Kaggle notebooks only have 2 cores?\n",
    "    df_split = np.array_split(df, n_cores)\n",
    "    pool = Pool(n_cores)\n",
    "    df = pd.concat(pool.map(func, df_split))\n",
    "    pool.close()\n",
    "    pool.join()\n",
    "\n",
    "    return df\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example usage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Oh My God this corpus rock! I am a gud speler. Earthquake plese! ? YOLOhere is'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_tweet = \"OMG this corpora <strong>rocks</strong>!!!!! I am a gud speler. \"\n",
    "test_tweet += \"http://www.cool.com Earthquake plese?!?!!!!    😔😔 !?!?!? #YOLO\"\n",
    "test_tweet += \"here is pic.twitter.com/12345\"\n",
    "\n",
    "clean_all(test_tweet, correct_spelling=False, remove_stopwords=False, lemmatize=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test on a random sample of real tweets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>clean</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7305</th>\n",
       "      <td>@Jennife29916207 I was thinking about you today when I was reading about the wild fires</td>\n",
       "      <td>Jennife29916207 I was thinking about you today when I was reading about the wild fires</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4322</th>\n",
       "      <td>#GamerGate 'Our entire attempt to hijack your community destroy your industry and scam everyone in sight was just sarcastic'.\\n\\nDrop dead.</td>\n",
       "      <td>GamerGate'Our entire attempt to hijack your community destroy your industry and scam everyone in sight was just sarcastic'. Drop dead.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4589</th>\n",
       "      <td>@msnbc What a fucking idiot. He had a gun &amp;amp; a hatchet yet there were still no serious injuries. Glad police terminated him.</td>\n",
       "      <td>msnbc What a fucking idiot. He had a gun and amp a hatchet yet there were still no serious injuries. Glad police terminated him.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3974</th>\n",
       "      <td>#flood #disaster Burst Water Pipe Floods Apartments at NYCHA Senior Center - NY1: NY1Burst Water Pipe Floods A... http://t.co/w7SIIdujOH</td>\n",
       "      <td>flood disaster Burst Water Pipe Floods Apartments at NYCHA Senior Center NY1 NY1Burst Water Pipe Floods A.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3920</th>\n",
       "      <td>And please don't flood poor @RobertBEnglund's mentions. He's put in his work!</td>\n",
       "      <td>And please don not flood poor RobertBEnglund is mentions. He is put in his work!</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4445</th>\n",
       "      <td>@pmarca content is held hostage by network due to affiliation fees.</td>\n",
       "      <td>pmarca content is held hostage by network due to affiliation fees.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6642</th>\n",
       "      <td>Anyone missing their license plate? Two stolen ones found on terrorist's car... http://t.co/CWGCciw3V6</td>\n",
       "      <td>Anyone missing their license plate? Two stolen ones found on terrorist is car.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4305</th>\n",
       "      <td>Hellfire is surrounded by desires so be careful and donÛªt let your desires control you! #Afterlife</td>\n",
       "      <td>Hellfire is surrounded by desires so be careful and don t let your desires control you! Afterlife</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1886</th>\n",
       "      <td>Crushed it! https://t.co/EWnUnp8Hdo</td>\n",
       "      <td>Crushed it!</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5626</th>\n",
       "      <td>Top story: @ViralSpell: 'Couple spend wedding day feeding 4000 Syrian refugeesÛ_ http://t.co/a2TIIVNjDY see more http://t.co/fW2XIfJ6Ec</td>\n",
       "      <td>Top story ViralSpell 'Couple spend wedding day feeding 4000 Syrian refugees see more</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                             text                                                                                                                                    clean\n",
       "7305  @Jennife29916207 I was thinking about you today when I was reading about the wild fires                                                       Jennife29916207 I was thinking about you today when I was reading about the wild fires                                                \n",
       "4322  #GamerGate 'Our entire attempt to hijack your community destroy your industry and scam everyone in sight was just sarcastic'.\\n\\nDrop dead.   GamerGate'Our entire attempt to hijack your community destroy your industry and scam everyone in sight was just sarcastic'. Drop dead.\n",
       "4589  @msnbc What a fucking idiot. He had a gun &amp; a hatchet yet there were still no serious injuries. Glad police terminated him.               msnbc What a fucking idiot. He had a gun and amp a hatchet yet there were still no serious injuries. Glad police terminated him.      \n",
       "3974  #flood #disaster Burst Water Pipe Floods Apartments at NYCHA Senior Center - NY1: NY1Burst Water Pipe Floods A... http://t.co/w7SIIdujOH      flood disaster Burst Water Pipe Floods Apartments at NYCHA Senior Center NY1 NY1Burst Water Pipe Floods A.                            \n",
       "3920  And please don't flood poor @RobertBEnglund's mentions. He's put in his work!                                                                And please don not flood poor RobertBEnglund is mentions. He is put in his work!                                                       \n",
       "4445  @pmarca content is held hostage by network due to affiliation fees.                                                                           pmarca content is held hostage by network due to affiliation fees.                                                                    \n",
       "6642  Anyone missing their license plate? Two stolen ones found on terrorist's car... http://t.co/CWGCciw3V6                                       Anyone missing their license plate? Two stolen ones found on terrorist is car.                                                         \n",
       "4305  Hellfire is surrounded by desires so be careful and donÛªt let your desires control you! #Afterlife                                         Hellfire is surrounded by desires so be careful and don t let your desires control you! Afterlife                                      \n",
       "1886  Crushed it! https://t.co/EWnUnp8Hdo                                                                                                          Crushed it!                                                                                                                            \n",
       "5626  Top story: @ViralSpell: 'Couple spend wedding day feeding 4000 Syrian refugeesÛ_ http://t.co/a2TIIVNjDY see more http://t.co/fW2XIfJ6Ec     Top story ViralSpell 'Couple spend wedding day feeding 4000 Syrian refugees see more                                                   "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_test = train.sample(10)\n",
    "train_test['clean'] = train_test.apply(lambda x: clean_all(x['text']), axis=1)\n",
    "train_test[['text', 'clean']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clean both train and test dataframes. Save the final files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--- 2.3394715785980225 seconds ---\n",
      "--- 1.0346345901489258 seconds ---\n"
     ]
    }
   ],
   "source": [
    "start_time = time.time()\n",
    "train = parallelize_dataframe(train, clean_dataframe)\n",
    "train['text'] = train['clean']\n",
    "train = train.drop(columns=['clean'])\n",
    "\n",
    "print(\"--- %s seconds ---\" % (time.time() - start_time))\n",
    "\n",
    "start_time = time.time()\n",
    "test = parallelize_dataframe(test, clean_dataframe)\n",
    "test['text'] = test['clean']\n",
    "test = test.drop(columns=['clean'])\n",
    "print(\"--- %s seconds ---\" % (time.time() - start_time))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "train.to_csv('train_df_clean.csv', index=False)\n",
    "test.to_csv('test_df_clean.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
